In normal usage, software applications interact with other systems or users. This interaction can occur through Input/Output (I/O) devices coupled with a hardware device running the software application or as communications from other software applications. The I/O devices provide information from a variety of sources including networking devices, machinery controlled by the software applications, and hardware devices used by users of a system to facilitate data input and output to and from the system. Examples of such hardware devices are keyboards, displays and mice. These hardware devices are for real-time input and output of typed and graphical information with a software application.
There are times when real time user interaction with a software application is not desired. Reasons for this include the error prone nature of user interaction as well as the high cost of providing users to operate software. Thus, it is desirable to automate the operation of software applications in certain environments.
For example, as software applications are developed or troubleshot, changes are made to the software application. After these changes are made, it is frequently desirable to test the software application's functionality using such techniques as regression and functional testing to ensure that, as a result of the changes made, other aspects of the operation of the software application have not been adversely affected. To ensure the repeatability of the results of such testing, it is desirable to automate the process of performing regression and functional testing.
In addition to automating software testing, there are several other uses for removing real time interaction with software applications. For example, when developing a product demonstration for use in trade shows, it is desirable to be able to have a product perform an automated execution of a software application, without requiring user attendance, to provide a canned demonstration of the software application. Another example where automated operation of a software application is desirable is in the testing of systems by remote means. By executing locally stored tests from a remote machine, a remote user can determine potential problems with a local software application. A plethora of other scenarios exist where the automated interaction with a software application is desirable.
Currently, methods for automating the interaction with user interfaces of software applications operate by indicating pixel locations on a screen for an input action to be applied to the software application. First, while a user interacts with the software application, a separate recording application will record screen locations, in pixels, of mouse/key actions made by the user. Examples of such actions are mouse moves, mouse presses, mouse releases, mouse clicks and key actions such as presses or releases. This pixel and action information is stored in files in the system containing the software application. Then, during playback of these files by a replay application, the replay application follows the stored pixel information and applies the appropriate actions at the appropriate pixel location as indicated by the stored values. The underlying assumption in this method is that screen locations of programs/applications during playback will be the same screen locations as they were during the record stage. However, this is not necessarily the case. For example, when an application playback is performed by a different machine than the machine with which the record was performed, which uses a different display resolution, the actual software location on the screen may be at a different pixel location. This will likely cause an error to occur during the playback as an input action is applied to an incorrect window in the software application.